[CI] Extend and enhance Travis build matrix #1035

sebastianblum · 2018-01-08T14:35:05Z

Q	A
Branch?	2.0
Bug fix?	no
New feature?	no
BC breaks?	no
Deprecations?	no
Tests pass?	yes
Fixed tickets	-
License	MIT
Doc PR	-

Try to extend the travis ci build matrix to fail on skipped tests

dbu

thanks for looking into this!

dbu · 2018-01-08T16:16:57Z

.travis.yml

+      env:
+        - SYMFONY_VERSION=4.0.*
+        - FLYSYSTEM_VERSION=1.0.*
+        - SIMPLE_PHPUNIT_PARAMETERS=" --stop-on-skipped"


i think this is not doing what you think it does. it stops execution when a skipped test is encountered, but not treat it as failure. see https://travis-ci.org/liip/LiipImagineBundle/jobs/326390446

it looks like a --fail-on-skipped is considered as unimportant by sebastian, so we could either maybe do a complicated thing with a test listener to detect skipped tests, or use an environment variable that we define on the line that runs phpunit that we look at in those tests and if its present fail instead of skip.

dbu · 2018-01-08T16:18:03Z

.travis.yml

+    - php: 7.2
+      env:
+        - SYMFONY_VERSION=4.0.*
+        - FLYSYSTEM_VERSION=1.0.*


i like the approach we took here: https://github.com/php-http/HttplugBundle/blob/f034b168dde627de027c3d874410b10df2aa82af/.travis.yml#L56 - we define one dependencies variable and if any is set, have composer install those.

dbu · 2018-01-08T16:19:22Z

.travis.yml

    - env: SYMFONY_VERSION=dev-master
    - php: nightly
-
+
 before_install:
  - bash ./.travis/exec-before.bash

 install:
  - travis_retry composer require "symfony/symfony:${SYMFONY_VERSION}" $COMPOSER_FLAGS


while we are looking at these things, we could use dunglas/symfony-lock:^2 resp 3 and 4 to do this. the current code will hide missing dependencies on symfony components, as it installs all of symfony. the dunglas package conflicts with everything not in the right version instead, so the CI will show if there is a symfony component being used but not declared in require/require-dev.

sebastianblum · 2018-01-09T10:19:25Z

@dbu yes, I hoped that --stop-on-skipped return an exit code greater than zero.

At the moment I try to add the dependencies for the skipped tests locally.

sebastianblum · 2018-01-09T20:40:48Z

@dbu I fixed all tests. At the moment, Flysystem is included in composer.json so all are running.

I now try to remove the dependency, what do you think?

sebastianblum · 2018-01-09T20:46:35Z

I found a lot of wrong namespaces like here
https://github.com/liip/LiipImagineBundle/blob/2.0/Tests/Imagine/Filter/Loader/AutoRotateFilterLoaderTest.php

@dbu what do you think? can we run php-cs-fixer on the 2.0 branch?

dbu

this looks good to me!

I now try to remove the dependency, what do you think?

the build looks great, no skipped tests in the build with flysystem 👍
you did not add it in composer.json if i see correctly.

php-cs-fixer

as far as i can see, Tests/ is included in styleci, and styleci should find such problems. is styleci not checking for namespace mismatches? i think tests should be correctly namespaced and autoloadable.

sebastianblum · 2018-01-10T08:52:34Z

@dbu at the moment, it is included in require-dev see https://github.com/liip/LiipImagineBundle/blob/2.0/composer.json

we have two possibilities:

test case with composer update --no-dev and manually install simple-phpunit to see skipped tests
leave it as it is

what would you prefer?

dbu

now i don't get why i saw skipped tests earlier. or did you rebase on 2.0 meanwhile? anyways, i think that is good enough with flysystem in require-dev. some good clean ups in this branch, lets merge it.

sebastianblum · 2018-01-11T14:56:42Z

no @dbu, it is not ready at the moment. please give me one week more :)

https://travis-ci.org/liip/LiipImagineBundle/jobs/327718645
Here you can a test without the dev-dependencies and there should exist a lot skipped tests but without errors. We can discuss if we leave these tests. What do you think?

sebastianblum · 2018-01-14T14:39:59Z

@dbu now it looks better

composer plugin prestissimo:
aktiv: https://travis-ci.org/liip/LiipImagineBundle/builds/328729635
inaktiv: https://travis-ci.org/liip/LiipImagineBundle/builds/328712094
I can't see any advantage on travis. Maybe it is already included.
Should we remove it? what do you think?

build matrix:
see https://travis-ci.org/liip/LiipImagineBundle/builds/328729635

1. tests only the decencies in composer
1. tests with symfony/symfony:3.4.*
1. tests with symfony/symfony:4.0.*
...
1. tests without dev dependencies. Tests here should be skipped and not fail
1. only execute code coverage once and may fail if it takes too long

In my opinion it is ready for feedback.

robfrawley · 2018-01-21T10:37:47Z

@sebastianblum We intentionally limited the scope of the build matrix for 2.x after the difficulties we've had with the 1.x branch, mostly regarding the total time.

This PR almost doubles the previous total time of 10 minutes to 17. We need to tighten the scope of the build matrix in this PR and shoot for getting our times more in-line with what it was, or, alternatively, provide sound reasoning for each addition.

Remember, there are lots more Symfony versions to come and we like to test all supported minor releases, so the matrix will naturally become larger later; we will need headroom so that can happen without getting to the 1 hour 16 minute total time it takes for the 1.x series to complete its CI build. :-)

sebastianblum · 2018-01-21T10:55:01Z

@robfrawley good point. I removed two tests, they are not necessary in my opinion.

PHP 7.1 and PHP 7.2 without (symfony/symfony in a fixed version) are the most important tests.
Then the tests with fixed symfony/symfony version can run only in one php version.
The composer flags --prefer-lowest should also run in my opinion.
Also the check, that die require-dev are really optional.

PHP coverage runs now only once, also php 7.3 check.

Do you have more feedback?

robfrawley · 2018-01-21T11:21:37Z

@sebastianblum Yeah; that looks better and brought us within two minutes of the current 2.x branch. As for any other feedback, I only really glanced at the build matrix and travis times; I'll take a broader look at the other changes later today and let you know if I notice anything. Thanks for the PR!

sebastianblum · 2018-01-21T11:47:45Z

@robfrawley After this pr is merged, I will add another check for php-cs-fixer to the Build Matrix in pr #1040

I can add it to an existing test or create a new one. What would you suggest?

robfrawley

Aside from the change to .travis.yml requested, this PR looks great. Once those are addressed this will be ready for merge.

robfrawley · 2018-01-30T20:33:36Z

.travis.yml

+  - if [[ "$SYMFONY_VERSION" != "" ]]; then travis_retry composer require "symfony/symfony:${SYMFONY_VERSION}" --no-update $COMPOSER_FLAGS; fi;
+  - if [[ "$DEPENDENCIES" != "" ]]; then travis_retry composer require ${DEPENDENCIES} --no-update $COMPOSER_FLAGS; fi;
+  - if [[ "$STABILITY" != "" ]]; then composer config minimum-stability $STABILITY; fi
+  - if [[ "$ENABLE_CODE_COVERAGE" != "true" && "$TRAVIS_EVENT_TYPE" != "cron" ]]; then rm -f ~/.phpenv/versions/$(phpenv version-name)/etc/conf.d/xdebug.ini; fi;


Instead of rm -f ~/.phpenv/versions/$(phpenv version-name)/etc/conf.d/xdebug.ini you should use the phpenv configuration plugin instead; see "Preinstalled PHP extensions" (blog.travis-ci.com).

phpenv config-rm xdebug.ini

- fixed FlysystemTests - fix SymfonyFrameworkTest to use Kernel constants - use fly system depe in travis - test with --no-dev composer parameter - only install all symfony packages if version is specified - added matrix prefer-lowest - skip tests if browser-kit package is not available. - skip tests if package is not available. - skip doctrine/cache tests. - skip tests if symfony/form is missing - skip if twig/twig is missing. - added build to test dependencies - added missing use statement - skip if symfony/yaml is missing. - skip if doctrine/orm is missing. - Update .coveralls.yml - only 1 build with code coverage - removed hirak/prestissimo:^0.3 (no speed benefit) - running prefer-lowest also on the lowest php version - using phpenv to disable xdebug

robfrawley · 2018-01-30T21:55:44Z

Thanks @sebastianblum!

sebastianblum mentioned this pull request Jan 8, 2018

[Post-Processors] Replaced ProcessBuilder with Process and add additional tests #1025

Merged

dbu reviewed Jan 8, 2018

View reviewed changes

dbu reviewed Jan 9, 2018

View reviewed changes

dbu approved these changes Jan 10, 2018

View reviewed changes

lsmith77 added this to the 2.0.0 milestone Jan 20, 2018

sebastianblum mentioned this pull request Jan 20, 2018

[CS] Implement new php-cs-fixer rules and related code cleanup #1040

Merged

robfrawley requested changes Jan 30, 2018

View reviewed changes

robfrawley approved these changes Jan 30, 2018

View reviewed changes

sebastianblum and others added 2 commits January 30, 2018 16:48

fix builds without xdebug config ini and fix travis indentation

3253b6d

robfrawley changed the title ~~[2.0] extending travis build matrix~~ [CI] Extend and enhance Travis build matrix Jan 30, 2018

robfrawley merged commit c676dee into liip:2.0 Jan 30, 2018

sebastianblum deleted the extending-travis-build-matrix branch January 31, 2018 16:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Extend and enhance Travis build matrix #1035

[CI] Extend and enhance Travis build matrix #1035

sebastianblum commented Jan 8, 2018

dbu left a comment

dbu Jan 8, 2018

dbu Jan 8, 2018

dbu Jan 8, 2018

sebastianblum commented Jan 9, 2018

sebastianblum commented Jan 9, 2018 •

edited

Loading

sebastianblum commented Jan 9, 2018

dbu left a comment

sebastianblum commented Jan 10, 2018

dbu left a comment

sebastianblum commented Jan 11, 2018 •

edited

Loading

sebastianblum commented Jan 14, 2018

robfrawley commented Jan 21, 2018 •

edited

Loading

sebastianblum commented Jan 21, 2018

robfrawley commented Jan 21, 2018

sebastianblum commented Jan 21, 2018

robfrawley left a comment •

edited

Loading

robfrawley Jan 30, 2018

robfrawley commented Jan 30, 2018

[CI] Extend and enhance Travis build matrix #1035

[CI] Extend and enhance Travis build matrix #1035

Conversation

sebastianblum commented Jan 8, 2018

dbu left a comment

Choose a reason for hiding this comment

dbu Jan 8, 2018

Choose a reason for hiding this comment

dbu Jan 8, 2018

Choose a reason for hiding this comment

dbu Jan 8, 2018

Choose a reason for hiding this comment

sebastianblum commented Jan 9, 2018

sebastianblum commented Jan 9, 2018 • edited Loading

sebastianblum commented Jan 9, 2018

dbu left a comment

Choose a reason for hiding this comment

sebastianblum commented Jan 10, 2018

dbu left a comment

Choose a reason for hiding this comment

sebastianblum commented Jan 11, 2018 • edited Loading

sebastianblum commented Jan 14, 2018

robfrawley commented Jan 21, 2018 • edited Loading

sebastianblum commented Jan 21, 2018

robfrawley commented Jan 21, 2018

sebastianblum commented Jan 21, 2018

robfrawley left a comment • edited Loading

Choose a reason for hiding this comment

robfrawley Jan 30, 2018

Choose a reason for hiding this comment

robfrawley commented Jan 30, 2018

sebastianblum commented Jan 9, 2018 •

edited

Loading

sebastianblum commented Jan 11, 2018 •

edited

Loading

robfrawley commented Jan 21, 2018 •

edited

Loading

robfrawley left a comment •

edited

Loading